A two-level system model for score-based measurement

The normalized gain (or the g-factor) has been widely used in physics education community as an assessment measure for student performance. In particular, it allows researchers to compare different instructions using classes with different initial states. Systematic differences were identified by R. Hake with thousands of students, which show that classes, however different initially, tend to have similar values of g-factor when going through similar types of instruction, i.e., classes with traditional instructions often have systematically lower g’s than classes with research-based instructions. These results indicate that g-factor can remove the individualities of the classes and reflect cleaner information about the instruction. However, the question of why the g-factor has this feature is still not well understood. In this paper, a physical model for the g-factor is proposed based on the context dependency of learning that may explain the physics behind the g-factor. Three issues will be discussed in this paper. First, a model is developed to describe the score-based measurement of student knowledge and the change of the knowledge. This model leads to an explicit mathematical formulation of the possible dynamics of learning as reflected by the g-factor. Finally, other types of calculations are discussed and compared with the g-factor. I. Student knowledge structure as reflected by score-based measurement In general, a student’s knowledge on a particular concept topic can be considered with three categories: 1) scientifically correct knowledge, 2) scientifically incorrect knowledge, and 3) lack of knowledge. Depending on the nature of a particular measurement being conducted, the three components can result in different outcomes both independently and in combination. For example, if students’ mental models are analyzed as part of the assessment (e.g. Model Analysis), the correct and incorrect knowledge is then reflected in the probability distribution for a student to use the different types of mental models (which can be correct, partially correct, incorrect, etc.). The lack of knowledge often results in a larger probability on a null model. In the case of score-based measurement, the correct knowledge is measured with the successfulness of a student’s giving correct answers. In this case, the 2 and 3 categories are collapsed together to represent the student’s inability to provide correct answers, which is reflected by the complementary part of score (1-score). (A score used in this paper is always scaled to have a value between 0 and 1.) The measurement is then reduced to a single effective dimension. Apparently, many uncertainties are involved in such reduction of dimensions, but the benefit is the simplicity in application. Now let’s turn to the question of what can be represented by a measured score. Due to the complexity of cognitive process, it is very difficult to consistently attribute an actual mental construct to the score. The context features of test questions may activate different pieces of a student’s knowledge as well as random human errors. Before we can parse out all the different possible mechanisms in a problem-solving process, these act as “hidden” variables and create uncertainties in the measured results. To get around this problem, let’s consider the student’s knowledge as reflected in the measurement space, i.e., we categorize the pieces of knowledge to be correct and/or incorrect only based on the measured results and don’t attempt to probe why/how a particular result is produced. It is a very crude method but one can hardly go any further with score-based measurement. So in the score space, the knowledge will be identified with measured-correct, and measured-wrong (it may not be incorrect knowledge but is just not measured correct).